Research Article 


Feasible Implementation Strategies 
for Improving Vocabulary Knowledge 
of High-Risk Preschoolers: Results 
From a Cluster-Randomized Trial 


Elizabeth Spencer Kelley,?® R. Michael Barker,” © Lindsey Peters-Sanders,” Keri Madsen,° © 


Yagmur Seven,”® Xigrid Soto,?® 


Purpose: Many children begin school with limited 
vocabularies, placing them at a high risk of academic 
difficulties. The goal of this study was to examine the 
effects of a vocabulary intervention program, Story Friends, 
designed to improve vocabulary knowledge of at-risk 
preschool children. 

Method: Twenty-four early-childhood classrooms were 
enrolled in a cluster-randomized design to evaluate the 
effects of a revised Story Friends curriculum. In each 
classroom, three to four preschoolers were identified as 
having poor language abilities, for a total of 84 participants. 
In treatment classrooms, explicit vocabulary instruction 
was embedded in prerecorded storybooks and opportunities 
for review and practice of target vocabulary were integrated 
into classroom and home practice activities. In comparison 
classrooms, prerecorded storybooks included target 
vocabulary, but without explicit instruction, and classroom 
and home strategies focused on general language 
enrichment strategies without specifying vocabulary targets 
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to teach. Intervention activities took place over 13 weeks, 
and 36 challenging, academically relevant vocabulary 
words were targeted. 

Results: Children in the treatment classrooms learned 
significantly more words than children in the comparison 
classrooms, who learned few target words based on 
exposure. Large effect sizes (mean d = 1.83) were evident 
as the treatment group averaged 42% vocabulary knowledge 
versus 11% in the comparison group, despite a gradual 
decline in vocabulary learning by the treatment group over 
the school year. 

Conclusions: Findings indicate that a carefully designed 
vocabulary intervention can produce substantial gains in 
children’s vocabulary knowledge. The Story Friends program 
is feasible for delivery in early childhood classrooms and 
effective in teaching challenging vocabulary to high-risk 
preschoolers. 

Supplemental Material: https://doi.org/10.23641/asha. 
13158185 


dictors of later reading ability (National Early Liter- 

acy Panel, 2008; National Institute of Child Health 
and Human Development, 2000). In particular, vocabulary 
knowledge is a strong contributor to reading comprehension 
(Language and Reading Research Consortium & Logan, 
2017; Quinn et al., 2015). Children with limited language 
skills in preschool and kindergarten are at a high risk of 


| | arly language skills are important and reliable pre- 
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later reading failure and diagnosis with reading disability 
(Catts et al., 2002; Scarborough, 1998; Sénéchal et al., 2006), 
and children with limited vocabulary knowledge are likely 
to have comprehension deficits (Elwér et al., 2013; Nation 
et al., 2010). 

Although substantial efforts have been devoted to im- 
proving oral language skills of young children, there continues 
to be concerning evidence that many early childhood settings 
do not provide adequate support for early language develop- 
ment (Carta et al., 2014; Dickinson, 2011; Greenwood et al., 
2013; Wright, 2012). In a descriptive study of 65 early child- 
hood classrooms, less than 10% of observed intervals involved 
language-focused teacher talk, and even fewer intervals in- 
volved vocabulary-focused teacher talk (Carta et al., 2014). 
Similar findings were reported by Dwyer and Harbaugh 
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(2018); in low-income preschool classrooms, teachers occa- 
sionally provided explicit vocabulary instruction when 
delivering science content but never during time focused on 
reading. Wright (2012) reported that explicit vocabulary in- 
struction occurred only rarely across kindergarten classrooms 
and less often in classrooms serving low-income students. 


Vocabulary Intervention in Early Childhood 


There is compelling evidence that well-designed vocab- 
ulary interventions can improve the oral language skills of 
young children. In a meta-analysis of 67 vocabulary inter- 
vention studies with preschool and kindergarten children, 
Marulis and Neuman (2010) reported large effects for 
vocabulary learning (g = 0.88), although effects were much 
smaller for at-risk children from low-socioeconomic status 
families (g = 0.77) than for children from middle- to high- 
socioeconomic status families (g = 1.35). In many studies, 
vocabulary interventions have been delivered in shared read- 
ing activities with embedded, explicit lessons and have 
improved vocabulary knowledge of children with large 
effect sizes (Coyne et al., 2007; Justice et al., 2005; Loftus 
et al., 2010). 

Although effect sizes have been moderate to large 
across studies, an examination of vocabulary gains in 
terms of number or percentage of words learned reveals 
only modest increases in vocabulary knowledge. For exam- 
ple, in Justice et al. (2005), children learned 2.3—2.7 new 
words on average of the 30 words included in the explicit 
teaching condition, and Penno et al. (2002) reported gains 
of 2.7 of 10 explicitly taught words. Coyne et al. (2010) re- 
ported gains of 23% of possible word points, and Neuman 
et al. reported gains of between 9% and 14% of words 
taught per unit (Neuman et al., 2011). In studies of kinder- 
garten children with developmental language disorders, 
Storkel and colleagues (Storkel et al., 2019, 2017) reported 
learning of 17%-—30% of words when dosage and frequency 
of exposure were optimized. When vocabulary intervention 
studies have included a delayed posttest, treatment effects 
are often maintained (Loftus et al., 2010; Neuman et al., 
2011), but children retain only a portion of learned words 
(e.g., 70% in Coyne et al., 2007; 47% in Kelley et al., 2015). 
Although these gains are significant and meaningful, there 
is a need for interventions that can produce more robust 
effects, especially for at-risk children with limited oral lan- 
guage skills. 


Increasing Effects on Vocabulary Knowledge 


To increase effects of interventions on vocabulary 
knowledge, one approach is to simply teach more words. 
However, research has yet to resolve which words to teach 
or the optimal number of words to target simultaneously. 
Some intervention programs teach a wide variety of word 
types and purposefully include a substantial proportion of 
words that children are likely to know already. For exam- 
ple, Neuman et al. (2011) selected sets of vocabulary targets 
in which more than 40% of the words were likely to be 
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familiar to preschool children. In each 24-day unit, teachers 
targeted between 50 and 80 words; thus, approximately 
20-30 new words were taught in a month. Other programs 
have focused on challenging words unlikely to be known by 
preschool children. These programs often have followed 
guidelines by Beck et al. (2013) that argue that explicit in- 
struction should focus on sophisticated vocabulary words 
with high utility for conversation and later reading. Within 
these programs, there is some variability in the number of 
words selected for instruction with a range typically be- 
tween two to six per book and one book per week (e.g., 
Coyne et al., 2010; Loftus et al., 2010). 

Outcomes of vocabulary interventions also can be 
improved by increasing how many of the taught words are 
learned. Characteristics of effective vocabulary interven- 
tion include explicit instruction, active engagement, and re- 
peated exposure to instructional targets. In comparison to 
indirect teaching or incidental exposure, explicit teaching 
produces much larger gains on vocabulary learning (Coyne 
et al., 2007; Justice et al., 2005). Although some vocabu- 
lary learning takes place from incidental exposure (e.g., re- 
ceptive knowledge of object labels; O’Fallon et al., 2020), 
children are rarely able to define or produce words for which 
they have had only incidental exposure (Coyne et al., 2009; 
Goldstein et al., 2016). Active engagement and giving chil- 
dren many chances to respond help children learn new 
words (Dickinson & Smith, 1994; Senéchal, 1997). Repeated 
exposure to instructional targets by repeated readings of the 
story or discussion of the vocabulary words increases the 
likelihood that children will learn those words (Flack et al., 
2018; Penno et al., 2002; Senéchal, 1997; Stahl & Fairbanks, 
1986). Repeated exposures can also be achieved within 
embedded lessons; in a recent study of a vocabulary inter- 
vention for kindergarten children with language disorders, 
Storkel et al. (2017) identified 36 as the number of exposures 
to target vocabulary necessary to provide maximum gains. 


Implementation in Educational Settings 


One challenge to effective oral language intervention 
is implementation in authentic education delivery settings 
(Foorman & Moats, 2004). High-fidelity implementation is 
critical to the effectiveness of interventions and, consequently, 
the improvement of educational outcomes (Justice et al., 
2008; Kaderavek & Justice, 2010; O’Donnell, 2008). Effi- 
cacy studies in which researchers are involved to ensure 
high implementation fidelity allow interventions to be im- 
plemented with precision and rigorous standards. However, 
when interventions move to teacher-implemented settings, im- 
plementation fidelity can be reduced (Hulleman & Cordray, 
2009). To address this challenge, it is important to consider 
feasibility of implementation from the earliest stages of de- 
velopment and identify flexible, effective strategies that are 
useful and feasible for teachers and families in early child- 
hood settings. 

To increase feasibility of implementation in early child- 
hood classrooms, interventions have often been designed 
to fit into existing routines and practices of early childhood 
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classrooms. Many successful interventions have taken ad- 
vantage of the widely used practice of shared storybook 
reading (e.g., Beck & McKeown, 2007; Coyne et al., 2007; 
Dickinson et al., 2019). Other common routines of the early 
childhood classroom (e.g., rotations through learning cen- 
ters, morning meetings, teacher-led small group activities) 
provide opportunities to deliver practice or repeated expo- 
sure to instructional targets. For example, Neuman et al. 
(2011) included journal writing activities as part of a vocab- 
ulary intervention program. Hadley et al. (2019) embedded 
vocabulary instruction into directed play activities and re- 
ported large treatment effects. 

Because the language experiences of young children 
at home are important contributors to language and literacy 
development (Dickinson & Tabors, 1991), effective interven- 
tions can extend beyond the classroom and include the fami- 
lies of young children. Similar to the classroom, feasibility 
of implementation can be improved by consideration of 
contexts, practices, and routines of families. Intervention 
programs that are carefully designed can be effective for 
families with limited resources (e.g., Smith et al., 2005; Soto 
et al., 2020). 


Story Friends: Theory of Change and Previous Studies 

The Story Friends program is a supplemental vocabu- 
lary program designed for implementation in early childhood 
classrooms. The theory of change for the Story Friends 


Figure 1. Theory of change for the Story Friends program. 


program was formulated through previous research and an it- 
erative development process informed by multiple stake- 
holders (see Figure | for theory of change). Contextual 
factors (e.g., demographics of children and teachers) as 
well as characteristics of preschool classrooms and instruc- 
tion guided decisions about the design and delivery of instruc- 
tion. The storybook context was selected based on previous 
research for vocabulary instruction and as a practice that 
would be readily accepted by preschool teachers and incor- 
porated into classroom routines. We designed an auto- 
mated format for delivery of instruction, with prerecorded 
storybooks and vocabulary lessons, to be easy to implement 
by educational staff and not requiring extensive preparation 
or training. To increase the potential for high-fidelity imple- 
mentation, we also tailored the structure and length of the 
sessions to a typical 15-min, small-group center rotation. 
Key stakeholders were engaged in both the design and re- 
visions to the program. By focusing on feasibility of imple- 
mentation, this development approach was designed to 
produce an intervention that would be readily adopted and 
sustained in a variety of early childhood settings (Kelley & 
Goldstein, 2015). 

The Story Friends program has been developed and 
evaluated in a series of studies (Summarized in Kelley & 
Goldstein, 2015). The findings from early efficacy studies 
using single-case experimental designs (Kelley et al., 2015; 
Spencer et al., 2012) and cross-site replications (Greenwood 
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et al., 2016) informed revisions and refinements to the pro- 
gram, which was then evaluated in a cluster-randomized 
trial in 32 classrooms with 195 participants (Goldstein et al., 
2016). In the cluster-randomized trial, large, significant treat- 
ment effects were observed for the learning of target vocabu- 
lary words. Participants in the treatment classrooms learned 
an average of 3.4 word points of a possible 12 per unit, with 
large effect sizes of 0.70 (Cohen’s f*). Fidelity of implemen- 
tation was high, and teachers rated the program highly for 
acceptability and feasibility of use. 

In the randomized controlled trial (RCT), participants 
in the experimental condition learned an average of 28.3% 
of target vocabulary. Although this percentage is higher 
than in other similar studies of vocabulary intervention, it 
is consistent with the common finding that children learn 
only a small percentage of vocabulary words taught. Thus, 
we resolved that the next iteration of the Story Friends 
program should focus on instructional strategies that could 
improve student learning. Two specific areas of improve- 
ment were targeted: increasing the number of words taught 
(Peters-Sanders et al., 2020) and providing opportunities 
for review and practice in the classroom (Seven et al., 2020) 
and at home (Soto et al., 2020). As with previous Story 
Friends work, the development efforts continued to focus 
on high-fidelity implementation in authentic educational 
settings. 


The Current Study 


The purpose of the current study was to evaluate the 
effects of the revised Story Friends program on vocabulary 
knowledge of preschool children. We sought to address 
two important challenges identified in previous research: 
the modest number of words learned and the feasibility of 
implementation in educational settings. Our previous studies 
have established the efficacy of the automated listening 
centers. In this study, the revised Story Friends program 
doubled the number of vocabulary targets per book and 
included materials to facilitate review and practice strate- 
gies across the day in the classroom and at home. 

The following research questions were addressed: 


1. What are the effects of the Story Friends program on 
preschoolers’ learning of target vocabulary words? 


2: Are observed treatment effects moderated by pre- 
intervention language scores or intervention dosage? 


3: To what extent is vocabulary knowledge retained af- 
ter intervention? 


4. To what extent is the program implemented as in- 
tended, and what are parent and teacher ratings of 
the program? 


Method 
Experimental Design 


This study represents a Hybrid Type 2 study (Curran 
et al., 2012) in which we combined an examination of 
both treatment efficacy and implementation of the Story 
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Friends intervention package. The foundation of the effi- 
cacy component of our study was the automated listening 
center conditions that sought to control the dosage of expo- 
sure to prerecorded story books with and without explicit 
vocabulary instruction. The implementation component 
focused on strategies that could be implemented flexibly 
across the day in the classroom and at home. Materials 
and training sought to facilitate those interactions between 
children and adults, but dosage varied largely under the 
control of the adults. 

The study used a cluster-randomized design with chil- 
dren nested in classrooms. Preschool classrooms were ran- 
domly assigned to the treatment or comparison condition 
using ranked pairs. Participating classrooms were rank- 
ordered by research staff familiar with the classrooms on 
three criteria: (a) the extent to which the classrooms were 
responsive and facilitative of the Story Friends program, 
as well as general administrative infrastructure and re- 
sources; (b) proportion of low-income families served at 
the center; and (c) the quality of the classroom structure 
including the presence of planned activities and routines. 
Using the ranked list, classrooms were paired (e.g., 1 and 2, 
3 and 4) and members of each pair were randomly assigned 
to treatment or comparison conditions. 


Setting 


Participants were teachers and children in 24 early 
childhood classrooms in two states (seven classrooms in 
Missouri and 17 in Florida). Classrooms selected served 
primarily children from families with low incomes and in- 
cluded full-day and half-day programs in session 4-5 days 
per week; 19 classrooms were 5 days per week, and five 
classrooms were 4 days per week. 


Participants 


Informed consent documents were shared with fami- 
lies in the 24 classrooms. Because the family strategies were 
available only in English, children in families who did not 
speak some English were not included. We used teacher 
report to identify families who spoke sufficient English to 
participate. We sought to identify children with oral language 
skills below age expectations who could benefit from the 
Story Friends intervention. Based on our previous research, 
we determined that children with low to below-average 
oral language skills were good candidates, whereas children 
with very limited oral language were unlikely to learn from 
listening to stories and instruction that they did not fully 
comprehend. 

For participants who returned signed consents, we 
considered information from the Oral Language screening 
measures of the Individual Growth and Development Indi- 
cators (IGDIs; Bradfield et al., 2014); a measure of recep- 
tive vocabulary, that is, the Peabody Picture Vocabulary 
Test—Fourth Edition (PPVT-4; Dunn & Dunn, 2007); and 
an omnibus measure of oral language, that is, the Clinical 
Evaluation of Language Fundamentals Preschool—Second 
Edition (CELF-P2; Wiig et al., 2004), to identify participants 
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Figure 2. CONSORT diagram showing the flow of participants throughout the experiment. 


Assessed for eligibility (1 = 239) 


Assigned to Treatment (n = 48) 


Discontinued Intervention (n = 7; 
left program) 


Excluded (n = 147) 
- Not meeting inclusion criteria 
(n = 136) 
- Other reasons (m7 = 11; did not 
assent to testing) 


Assigned to Comparison (n = 44) 


Discontinued Intervention (n = 1) 


Analyzed (n = 41) 


Analyzed (n = 43) 


(see CONSORT table, Figure 2). First, all eligible children 
completed the two IGDI Oral Language screening measures: 
Picture Naming and Which One Doesn’t Belong. IGDIs 
were administered individually to children using tablet com- 
puters. On the Picture Naming IGDI, children were asked 
to verbally label a set of 15 pictures. On the Which One 
Doesn’t Belong IGDI, children were presented with three 
pictures (e.g., bus, butterfly, and car) and asked to select 
the picture that did not belong. Children who scored below 
benchmark on either of the two measures remained in the 
pool of potential candidates. 

Next, children were given the PPVT-4 and CELF-P2. 
Our inclusionary criteria were standard scores on either the 
PPVT-4 or CELF-P2 between 0.5 and 2 SDs below the 
normative mean (70-92). In each classroom, three to four 
children who met these criteria were selected as participants. 
In most classrooms, the first children who met these criteria 


were selected. In some cases, when more than four children 
scored in this range, we asked teachers to give input on 
which children would be the most appropriate candidates 
(e.g., children with regular attendance). In rare cases, we 
extended the range slightly higher to identify a sufficient 
number of children in a classroom (i.e., two children were 
included with scores between 93 and 106). Table 1 includes 
sample characteristics. 

The selected group of participants included 84 children, 
43 boys and 41 girls, with an average age of 54 months 
(range: 45-63 months). Families were asked to complete a 
brief survey to provide demographic information; 75% of 
families returned the survey, but not all families completed 
all items. The majority of children were Black (47%, n = 30) 
or White (21%, n = 13); the remaining children were re- 
ported to be Asian, American Indian, Native Hawaiian, 
or Other Pacific Islander; 30% were reported to be Hispanic 
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Table 1. Sample characteristics by experimental groups. 


Group 

Treatment Comparison 
Variable ~ M(SD) — - M(SD) 
Children (n) 4 43 
Age at start (months) 55.32 (4.30) 53.79 (4.29) 
CELF-P2 pretest 80.45 (8.61) 79.72 (10.02) 
PPVT-4 pretest 89.90 (8.44) 87.72 (7.23) 
PPVT-4 posttest 93.32 (9.85) 92.23 (9.16) 


Note. CELF-P2 = Clinical Evaluation of Language Fundamentals 
Preschool—Second Edition (Wiig et al., 2004); PPVT-4 = Peabody 
Picture Vocabulary Test—Fourth Edition (Dunn & Dunn, 2007). 


or Latino; and 35% indicated that a language other than 
English was spoken in the home, with 17% of families 
reporting that Spanish was spoken. Three families reported 
that their child had an individualized education plan, and 
eight families reported having concerns about their child’s 
development. 

Of the families who reported annual household in- 
come (1 = 60), 30% reported income below $20 000, 47% 
reported income between $20 000 and $50 000, and 23% 
reported an income of more than $50 000. Just 32 families 
provided information about education of the primary care- 
giver, which ranged from some high school to a graduate 
or professional degree, whereas 53% of primary caregivers 
had education of high school or less, 19% had some college 
or an associate’s degree, 16% had a bachelor’s degree, and 
13% had a graduate or professional degree. 


The Story Friends Program 


In the current study, the Story Friends program in- 
cluded three components: automated listening centers, 
classroom strategies, and home strategies. Procedures 
and materials for each component are described below (see 
Supplemental Material S1 for figure of listening center ar- 
rangement, example story books, classroom, and home 
strategy materials). Each classroom was provided with a 
binder that included a teacher’s manual and attendance and 
fidelity checklists. 


Study Activities 

Teachers and other educational staff were responsi- 
ble for delivering the intervention. Research staff assisted 
teachers by providing materials, helping with scheduling, 
and troubleshooting equipment. Research staff were re- 
sponsible for administration of all assessments and con- 
ducted periodic observations of the classroom and 
listening centers. Research staff delivered materials for 
each unit (approximately monthly). 


Introduction to the Program 
Teachers and other educational staff participated in 
a brief (less than an hour) meeting with research staff to 
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learn about the program. These meetings were conducted 
with small groups of teachers or individual teachers at con- 
venient times. Research staff used the teacher’s manual to 
guide a conversation that included a brief overview of the 
purpose of the program, procedures for the listening cen- 
ters, use of classroom strategies, and delivery of home strat- 
egies. Research staff and teachers also discussed strategies 
for implementation, such as finding a location and time for 
the listening center. 


Treatment Classrooms 

Automated listening centers. The primary instruction 
of the Story Friends program was delivered in small-group 
listening centers. Listening center materials included sets 
of Forest Friends storybooks, headphones, an mp3 player 
with Story Friends audio files, and a splitter. The Forest 
Friends book series includes 13 books: one introductory 
book, nine instructional books, and three review books. 
The books are organized into units of three instructional 
books and a review book. Each book features the same 
animal characters with colorful illustrations and rhyming 
story text. The books are 9-11 min long. Children lis- 
tened to prerecorded storybooks with embedded lessons 
for four challenging vocabulary words, doubling that of 
our previous RCT. Challenging vocabulary words were 
19 verbs (e.g., agree, protect), 16 adjectives (e.g., powerful, 
delighted), and one noun (i.e., accident). Words were se- 
lected to be new words for preschool children with limited 
vocabulary, to be relatively high-frequency words that 
would likely occur again in conversations with adults, and 
to be able to be taught well in the prerecorded lessons (1.e., 
could be defined in simple, child-friendly terms). 

In the instructional books, each vocabulary word 
was taught in two embedded lessons, first when the word 
occurred in the story text and the second immediately after 
the story ended. For the challenging vocabulary words, 
each embedded lesson included explicit teaching of the 
word and a simple definition, a supportive story context, 
child-friendly examples, and multiple opportunities to re- 
spond. Sample lessons are provided in Table 2. The pattern 
of instructional language was consistent across embedded 
lessons; some variation was necessary to accommodate 
teaching of different vocabulary targets. In the instructional 
books, each target vocabulary word was presented 13-— 
15 times and the simple definition was presented 7-8 times. In 
each review book, the target vocabulary word was presented 
3 times and the definition was presented 2 times. Thus, a child 
who received the intended dosage of three listens to each in- 
structional and review book heard each target vocabulary 
word 48—54 times and each definition 27—30 times. 

Children listened in small groups of three to four chil- 
dren under headphones with an adult facilitator present. 
The facilitator was not expected to provide additional in- 
struction; instead, the role of the facilitator was to help 
guide children at the listening center (e.g., keep headphones 
on, stay on the correct page). In some classrooms, the teacher 
was the facilitator. In other classrooms, teacher assistants 
or classroom volunteers facilitated the listening center. 
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Table 2. Sample explicit, embedded lessons for vocabulary words. 


Lesson Example 


Explanation 


Embedded lesson — 
during story 


Pablo must prepare to leave. He has to get ready. 
Prepare. Say prepare. [pause] Prepare means to 
get ready. Tell me, what word means to get ready? 
[pause] Prepare. Great work! | bet you have to 
prepare to go to school in the morning. Maybe 
you get ready by eating breakfast and getting 


Each embedded lesson includes multiple opportunities to 
respond, indicated by pauses in the script. Children 
were given an opportunity to repeat the word, say the 
word in response to the definition, and provide the 
definition. Story context, a simple definition, and multiple 
child-friendly examples were included in each lesson. 


dressed. Now lift the flap. [Picture of a boy putting 
school supplies in his backpack] Look at this boy 
putting his supplies in his backpack. He is getting 
ready to leave school. He is preparing to go home. 
Tell me, what does prepare mean? [pause] To get 


ready. Awesome job! 
Embedded lesson — 
after story 


Look at the picture of the woman preparing food. 
[Picture of a woman chopping vegetables] She is 
getting ready for dinner. She is preparing food for 
her family. Say prepare. [pause] Prepare. Tell me, 


The second embedded lesson was presented immediately 
after the story and included an additional child-friendly 
context. Children were given an opportunity to repeat 
the word and to provide the definition. 


what does prepare mean? [pause] To get ready. 


Way to go! 


Classroom strategies. The purpose of the classroom 
strategies was to encourage teachers to provide opportu- 
nities for instruction and practice of targeted vocabulary 
words during daily classroom routines. We anticipated that 
children’s learning would be increased if teachers and 
children used the words taught in the automated listen- 
ing centers in meaningful classroom contexts. Classroom 
strategies were developed in an iterative process using in- 
formation from teacher focus groups and development 
studies (Seven et al., 2020). The classroom strategies were 
designed to be easy to use, adaptable, and readily incor- 
porated into typical classroom routines with minimal de- 
mands on teachers. 

The classroom strategy materials included Teacher 
Prompt Cards, a Story Friends Weekly Word Chart, and a 
Story Friends Review Board. The goal of the Teacher Prompt 
Cards and the Story Friends Weekly Word Chart was to 
increase the frequency of practice opportunities for the 
words taught that week. For each book, teachers received 
a prompt card for the four words taught that week. The 
prompt card included the taught, child-friendly definition 
and five examples of sentences that teachers could use to 
practice the words in daily classroom conversations. For 
example, for the word delighted, the prompt card included 
sentences such as “I am delighted to see you doing a good 
job.” and “You are smiling because you are delighted.” 
The Story Friends Weekly Word Chart was a poster and 
Velcro-backed cards for each word with the taught definition 
and an illustration from the book. Teachers could track when 
a teacher or child used the word in classroom conversations 
by adding a card to the column for that vocabulary word. 
The Story Friends Review Board was designed to function 
as a “word wall” to promote review and practice of all the 
words taught. Teachers received a poster and Velcro-backed 
cards for each word in the Forest Friends series with the 
taught definition, an illustration from the book, and a pho- 
tograph of a real-life context. 


Teachers learned about the classroom strategies dur- 
ing training and from the teacher’s manual. We provided 
recommendations for how teachers would use the materials, 
including guidance about how often to practice words. For 
example, we suggested that teachers use the Story Friends 
Weekly Word Chart to practice each word at least once per 
day and use the Story Friends Review Board to review all 
previously taught words once a week. However, teachers 
were encouraged to use the materials in the ways that worked 
best for their classroom routines. The intention was to provide 
teachers with a variety of materials and strategies that could 
be readily incorporated into classroom routines and easily 
shared with families. 

Home strategies. The goal of the home strategies was 
to encourage families and children to talk about and use 
the targeted vocabulary words at home, and they were 
intended to be feasible for use by families as part of daily 
conversations or brief activities. Similar to the classroom 
strategies, we anticipated that learning would be improved 
if children had frequent opportunities to hear and use the 
words. The home strategies and materials were developed 
in a series of single-case design studies (Soto et al., 2020). 

Home strategy materials included a family training 
video, take-home materials, and communication via Bloomz, 
a web-based communication platform. The family training 
video presented a brief overview of importance of vocabu- 
lary in children’s future academic success and provided 
information about the Story Friends program. Examples 
of the home strategy materials were presented along with 
video examples of an adult practicing the target vocabu- 
lary. The video was approximately 8 min long and was 
shared with families in a number of ways. Families could 
receive a link to the video via Bloomz, and the link was 
included on flyers given to parents and posted in the class- 
room. The video also was downloaded onto tablet com- 
puters and shared with families at pickup and drop-off and 
home visits. 
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Take-home materials were designed to encourage 
families and children to use the taught vocabulary words. 
Materials included stickers, necklaces, and family diary 
forms. Stickers included an illustration from the story and 
a prompt for one target word (e.g., “Ask me what brave 
means’). Necklaces included a laminated card with story 
illustrations for each of the four words taught that week 
and the words and taught definitions printed on the back. 
Family diary forms were laminated cards designed to hang 
on doors or the rear view mirror of cars. Each form in- 
cluded 12 boxes that families could mark each time they 
practiced a vocabulary word. 

To facilitate communication with families, teachers 
and parents were encouraged to set up a Bloomz account 
that could be accessed on a computer or a phone with In- 
ternet connection. Teachers created a virtual classroom 
and shared an electronic link with families. Families who 
joined the virtual classroom could receive automated daily 
reminders that said, “Don’t forget to practice your chil- 
dren’s words today.” Teachers also used Bloomz to share 
a library of brief video clips designed to extend the vocabu- 
lary instruction for each book. Three videos were created 
for each book; each video included a storybook illustration 
or photograph related to a target vocabulary word. The 
first video included prompts for yes/no questions related to 
the word’s meaning (e.g., When you watch your favorite 
cartoon, are you delighted?). The second video included 
prompts for activities or open-ended questions related to 
the target vocabulary (e.g., Ask your child to give you a 
big smile. Say “Wow! You look delighted. You look really 
happy.). The third video included prompts for using the 
target vocabulary word in a sentence (e.g., “Ask your child 
to use delighted in a sentence. If your child does not know a 
sentence say”; “The boy was delighted to eat yummy cake! 
Delighted means really happy!’). Then, they must work to- 
gether to make another sentence. 

The original plan was for families to receive text mes- 
sages and access video clips via Bloomz. However, within 
the first month of intervention, it was apparent that few 
families chose to enroll in Bloomz. To provide an alterna- 
tive to Bloomz, we prepared brief e-mail templates with 
prompts and links to the video clips that teachers could 
send to families. 

Classroom teachers distributed the home materials. 
Similar to the classroom strategies, teachers received training 
and information in the manual that described the intended 
use, as well as strategies for distributing the materials. Re- 
search staff worked with teachers to ensure that all families 
had an opportunity to watch the training video and encour- 
aged teachers to distribute the home materials to families 
by sending home stickers or necklaces each day. However, 
teachers were free to adapt use of home materials according 
to what worked best for their classroom. 


Comparison Classrooms 

Rather than a business-as-usual control condition, the 
comparison condition in the current study was designed 
to provide an active control and robust comparison to the 
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Story Friends program. Teachers in comparison classrooms 
received training and materials designed to promote overall 
language enrichment, as opposed to the emphasis on ex- 
plicit vocabulary instruction in treatment classrooms. In 
the comparison classrooms, children participated in small- 
group listening centers and listened to Story Friends books, 
but without explicit, embedded lessons. Thus, children in 
the comparison classrooms were exposed to the same targeted 
vocabulary in the context of a story, but not augmented with 
explicit instruction. Classroom and home strategies empha- 
sized language enrichment via interactive, shared storybook 
reading. Teachers and families were provided with a brief 
video presentation, available in multiple formats (i.e., Power- 
Point presentation file, a handout, and a YouTube video) 
that taught interactive, shared reading strategies. Content for 
the presentation was adapted from research on dialogic read- 
ing (Whitehurst et al., 1988; Zevenbergen & Whitehurst, 
2003) and emphasized strategies such as asking questions 
and providing opportunities for children to respond dur- 
ing shared reading. Families in comparison classrooms 
were invited to participate in Bloomz to receive automated 
messages (e.g., reminders to read with their child). Similar 
to treatment classrooms, very few families participated in 
Bloomz. We created e-mail templates for teachers to send 
to parents that focused on interactive book reading. 


Outcome Measures 


The primary outcome measure was a researcher-created 
proximal measure: the Unit Vocabulary Test. The Unit Test 
assessed definitional knowledge of target vocabulary using a 
rigorous test of children’s vocabulary knowledge. The defini- 
tional task provides several advantages to a picture-pointing 
or other receptive measure. First, the definitional task assesses 
rich, decontextualized word knowledge that contributes to 
a variety of language skills (e.g., reading comprehension). 
Second, the definitional task has been widely used in previ- 
ous research, including our own, allowing for comparisons 
of treatment effects across studies. Third, the definitional 
task was a straightforward, feasible measure for use by 
teachers in future scale-up use of the program. Finally, 
the definitional task was most appropriate for the type of 
words targeted; most Story Friends vocabulary words (e.g., 
protect, brave, search) were not easily represented by pic- 
tures that could be reliably recognized by preschoolers. 

Participants were asked to respond to an open-ended 
definitional item (Tell me, what does lost mean?). If children 
did not respond correctly, a sentence-length prompt specific 
to the target vocabulary word was administered (Jn the story, 
the Forest Friends were lost. Lost means...). Responses were 
scored on a 3-point scale: 2 points were awarded for a 
complete definition, either the taught definition or a rea- 
sonable definition in the child’s own words, or for an accu- 
rate synonym; | point was awarded for a partial definition, 
for using the target word in a meaningful sentence or phrase, 
or for providing an example of the word; and 0 points were 
awarded for an incorrect, unrelated, or “I don’t know” re- 
sponse. Children who repeated the sentence-length prompt 


Downloaded from: https://pubs.asha.org Elizabeth Kelley on 11/13/2020, Terms of Use: https://pubs.asha.org/pubs/rights_and_permissions 


were given a score of 0. For example, for the word delighted, 
responses of “really happy,” “happy,” or “thrilled” received 
2 points. Responses of “J was delighted at that party” or 
“delighted to eat that cupcake” received | point. Responses 
of “be delighted” or “I like it” received 0 points. Our scor- 
ing approach was designed to provide a conservative esti- 
mate of children’s knowledge. For example, a child who 
responded “J like it” may have had some partial knowledge 
of delighted that was not captured in our scoring. 

At pretest, all 36 challenging vocabulary words were 
assessed. At the end of each unit, approximately monthly, 
learning of the 12 taught vocabulary words from that unit 
was assessed. The maximum score for each end-of-unit test 
was 24 (2 points per 12 words). To assess retention, six 
vocabulary words from Unit | were included at the Unit 3 
posttest. To identify items for the retention test, we exam- 
ined learning from Unit | and chose the words most fre- 
quently learned. On each posttest assessment, words were 
presented in two counterbalanced orders and participants 
were randomly assigned to each form. 


Postintervention Measures 


At the end of intervention, children were adminis- 
tered the PPVT-4. Although we did not anticipate treat- 
ment effects on the PPVT-4 because it did not test words 
that we taught, the measure provided an opportunity to 
describe the receptive vocabulary skills of children at the 
end of the study. 


Dosage and Implementation Fidelity 


In the current study, the Story Friends program was 
implemented as an intervention package that included the 
automated listening centers, classroom strategies, and home 
strategies. We set clear expectations for implementation of 
the listening centers; the intended dosage was for a child 
to listen 3 times in a week to each instructional or review 
book, for a total of 12 listens per unit (three listens each 
to three instructional books and one review book). During 
teacher training, research staff emphasized the importance 
of the repeated readings. Teachers were asked to complete 
daily attendance and procedural fidelity checklists. The fi- 
delity checklist included six items related to the delivery of 
the listening centers (i.e., each child had a book, each child 
had headphones, correct audio was played, entire audio 
was played, children were provided with reinforcement, 
and an adult was present in the listening center). The fidel- 
ity checklist is available in Supplemental Material S3. 

Research staff observed listening centers approximately 
once per month (~3 times per classroom) to assess procedural 
fidelity and to describe the listening centers. Observations 
lasted between 20 and 30 min and were scheduled at times 
convenient for the teachers. The observation form included 
items related to procedures (e.g., each child used headphones, 
correct audio was played, environment was quiet with few 
distractions). Observations also provided information to 
describe the listening centers, including use of positive 


feedback or additional instruction provided by the teachers 
and interruptions to the listening center in treatment class- 
rooms. The observation form is available in Supplemental 
Material S2. 

For the classroom strategies and home strategies, 
the manual and training provided general guidance on a 
variety of ways to review the target words and their mean- 
ings. However, we did not provide specific expectations 
(e.g., requiring a regularly scheduled classroom activity or a 
frequency of review). Thus, rather than fidelity, we sought 
to gather information about feasibility and use. For the 
classroom strategies, we asked teachers to self-report their 
practice events daily on the Story Friends chart, to take a 
picture of their chart each week, and to share it with us. 
Research staff also kept a record of any use of classroom 
strategies observed during fidelity observations as well as 
during other visits to classrooms or included in communica- 
tion with teachers. 


Social Validity Assessment 


At the end of the study, we gathered feasibility and 
implementation information from participating teachers 
and families through social validity surveys. Teachers rated 
their level of agreement with statements that pertained to 
their perception of the Story Friends program (e.g., “The 
intervention is a good way to address language delays”’), 
the ease of implementation (e.g., “The amount of time re- 
quired to use Story Friends is reasonable), and the frequency 
with which they utilized the classroom and home extension 
materials (e.g., “I sent vocabulary necklaces home weekly”). 
Teacher responded on a scale of 1 (strongly disagree) to 6 
(strongly agree). The survey included open-ended questions 
about experiences with the listening centers, classroom and 
home strategies, and what teachers might change about the 
program. 

For the home strategies, parents were asked to re- 
spond to similar types of questions rating their level of 
agreement about their perception of the Story Friends pro- 
gram (e.g., “My child is motivated to participate in the 
Story Friends home activities”; “I would be excited to use 
Story Friends in the future”) and about the frequency of 
implementation (e.g., “My child came home wearing Story 
Friends necklaces’). The survey included several open- 
ended questions about the home program (e.g., frequency 
of practice) and ways we could improve the home exten- 
sion component. 


Scoring Reliability 

A trained member of the research team scored all 
measures. For the CELF-P2 and PPVT-4, each protocol 
was scored by a primary scorer and then checked by a 
second scorer. The small number of disagreements was re- 
solved by a third scorer. 

For the Unit Vocabulary Test, a primary scorer at 
each site scored all measures and a second scorer indepen- 
dently scored approximately one third of all measures to 
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evaluate scoring reliability. Scorers were blinded to condi- 
tion (treatment, comparison) and to time (pretest, posttest). 
Detailed scoring guides for each unit were created to facili- 
tate reliable scoring. The scoring guide included detailed 
criteria for assigning a score as well as multiple examples 
for each word and each scoring category. Prior to scoring, 
research assistants at both sites reviewed scoring criteria 
and scoring guides and completed a training set. 
Item-by-item comparisons were conducted to determine 
agreement. Scoring reliability was calculated by dividing the 
total number of agreements by the sum of agreements plus 
disagreements and multiplying by 100. At pretest, there were 
a high number of 0-point answers (e.g., “I don’t know”), 
making scoring agreement more likely. Thus, agreement 
for pretest and posttest was examined separately. At pre- 
test, mean agreement was 97.8% (range: 96%-100%). At 
posttest, mean agreement was 92.9% (range: 87.5%-100%). 


Data Analysis 


The purpose of the current study was to examine the 
effects of the revised Story Friends program on the vocabu- 
lary knowledge of preschool children. The primary outcome 
was word score earned by each participant at pre and post for 
each book taught, derived from Unit Tests. For each book, 
children could earn a maximum of 8 points (2 points per word 
for four words). Each participant had a pre and post score for 
each book, for a total of 18 scores. Moreover, in addition to 
the repeated measures of pre and post across the nine books, 
the data were multilevel, with children nested within class- 
rooms. As such, we employed a multilevel modeling (MLM) 
approach to analyze the data that allowed for accounting 
of the clustering of data (i.e., children nested within class- 
rooms). This approach allowed us to analyze all available 
data without removing individual children or observations 
when one or more data points may have been missing. All 
data analyses were conducted using JMP Pro 14.3. 


Results 
Missing Data 


Prior to MLM analysis, missing data patterns, bro- 
ken down by intervention group, were examined using the 
Missing Data Pattern routine in JMP Pro 14.3. For the 
Story Friends treatment group, 35 of 41 students (85%) 
had complete data at pre and post. At post, one student 
was missing data for the last three books (7, 8, and 9), 
and a second student from a different classroom was 
missing data for the last six books (4, 5, 6, 7, 8, and 9). 
At pre, three students from two different classrooms were 
missing for all nine books, and a fourth student from a 
third classroom was missing data for the last four books 
(6, 7, 8, and 9). Overall, however, the missing data rate 
for the Story Friends treatment group was very low. Data 
were missing for only 40 of the 738 observations (5.4% of 
9 books x 2 observations each x 41 participants in the 
treatment group). 
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For the comparison group, 35 of 43 students (81%) 
had complete data at pre and post. Four students from two 
different classrooms were missing pre and post data from 
the last three books (7, 8, and 9), and four students from 
three different classrooms were missing pre and post data 
from six books (4, 5, 6, 7, 8, and 9). Again, however, the 
overall missing data rate was low. Data were missing from 
only 72 of the 774 observations (9.3% of 9 books x 2 obser- 
vations each x 43 participants in the comparison group). 

We conducted a series of ¢ tests to test for differences 
between children who were missing data and children with 
complete data on key variables, reported in the Appendix. 
There were no significant differences between the children 
with missing data and those with complete data on the IGDI 
Picture Naming, PPVT-4 prior to intervention, PPVT-4 
following intervention, or the average word scores across 
the nine books at pre- or postintervention, and effect sizes 
were small or less. There was a significant difference on 
the CELF-P2 prior to intervention, ¢(79) = —2.16, p < .05, 
with the children with complete data having a slightly higher 
score (M = 80.71, SD = 9.03) than children who had missing 
data (M = 74.36, SD = 9.25), representing a medium to 
large difference, d = 0.69. Importantly, there were no signif- 
icant differences in terms of word scores as a function of 
missing data. 


Data Screening and Descriptive Results 


Independent and dependent measures were evaluated 
to determine appropriateness for the proposed data analy- 
ses. A visual inspection of the distributions of the IGDI 
Picture Naming, CELF-P2, pre-intervention PPVT-4, pre- 
intervention word score, and postintervention word score 
indicated that all variables were approximately normally 
distributed. Means, standard deviations, medians, and skew 
statistics for these variables are presented in Table 3. Again, 
these statistics confirm the results of the visual inspection; 

a high degree of skew is indicated by skewness values less 
than —2 or greater than 2 (George & Mallery, 2016). 

Next, we conducted a series of between-subjects ¢ tests 
to test for differences between the treatment and comparison 
groups prior to intervention, reported in Table 4. There 
were no significant differences between treatment and 


Table 3. Distributional characteristics of independent and dependent 
variables. 


Variable M SD Mdn Skew 
CELF-P2 79.85 9.26 81 -0.10 
Pre PPVT-4 88.72 7.86 88 0.26 
Pre word score 0.36 0.40 0.22 1.65 
Post word score 2.04 1.87 1.28 1.02 


Note. CELF-P2 = Clinical Evaluation of Language Fundamentals 
Preschool—Second Edition (Wiig et al., 2004); PPVT-4 = Peabody 
Picture Vocabulary Test—Fourth Edition (Dunn & Dunn, 2007); Pre 
word score = pretest score per book out of 8 possible points; Post 
word score = posttest score per book out of 8 possible points. 
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Table 4. Group comparisons for independent variables. 


Treatment Comparison 
Variable M SD M SD t p d 
IGDI PN 47.4 2.43 47.29 1.71 -0.33 .74 0.07 
CELF-P2 79.95 8.48 79.77 10.00 -0.09 .93 0.02 


Pre PPVT-4 89.68 8.37 87.79 7.30 -1.10 .27 0.24 
Pre word score 0.40 0.42 0.31 0.38 -1.02 .31 0.23 


Note. |GDI PN = Individual Growth and Development Indicators 
(Bradfield et al., 2014); CELF-P2 = Clinical Evaluation of Language 
Fundamentals Preschool—Second Edition (Wiig et al., 2004); 
PPVT-4 = Peabody Picture Vocabulary Test—Fourth Edition (Dunn & 
Dunn, 2007); Pre word score = pretest score per book out of 8 possible 
points. 


comparison groups on the IGDI Picture Naming, CELF-P2, 
pre-intervention PPVT-4, or pre-intervention word score. 
There was a small effect size for the pre-intervention PPVT-4 
and the pre-intervention word score. Thus, we included 
pre-intervention PPVT-4 scores as a covariate in our subse- 
quent models. There was no need to include pre-intervention 
word score as a covariate, as this variable was an outcome 
in all of our models. 


Story Friends Effects on Vocabulary Learning 


A 2 x 2x 9 mixed between- and within-subjects anal- 
ysis was conducted within the MLM framework to deter- 
mine the impact of the Story Friends intervention on word 
scores. The following variables were modeled as fixed effects. 
The first factor was between subjects and evaluated differ- 
ences between the treatment and comparison groups. The 
second factor was within subjects and modeled participants 
scores pre- and postintervention. The final factor was within 
subjects and modeled differences between the individual 
books used in the study. Finally, as described previously, 
pre-intervention PPVT-4 scores were modeled as a covariate 
to control for the impact of the small differences we observed 
between the two groups at the beginning of the study. 

In addition to fixed effects, the random effect for dif- 
ferences between children and for differences between class- 
rooms were included in the analysis to account for the 
clustered structure of the data. The intraclass correlation 
(ICC) for differences between children was .250, indicating 
that 25% of the differences in word scores were attributable 
to individual children. The ICC for classroom was .063, in- 
dicating that 6.3% of the differences in word scores were at- 
tributable to differences between classrooms. Importantly, 
after accounting for these influences on word scores, an ad- 
ditional 68.6% of the variance was left to be explained by 
the fixed effects entered into the model next. 

The model containing fixed effects accounted for a 
total of 58.7% of the variance in observed word score out- 
comes. Pre-intervention PPVT-4 score was a significant pre- 
dictor of outcomes, b = 0.04, SE = 0.01, t = 3.78, p < .01, 
justifying its inclusion as a covariate in the model. Each of 
the main effects for treatment condition, book, and time 


were significant predictors. Moreover, all of the main ef- 
fects were qualified by significant two-way interactions for 
each possible combination of factors. Finally, there was a 
significant three-way interaction between condition, book, 
and time. These results are presented in Table 5. 

The estimated marginal means, controlling for pre- 
intervention PPVT-4 performance, for each group at pre- 
and postintervention across all nine books are presented in 
Figure 3. Each mean is presented with a 95% confidence 
interval. As clearly depicted in this figure, the significant 
three-way interaction was best characterized by the fact that, 
following treatment, children in the Story Friends treatment 
group showed significantly higher word scores, ranging from 
4.43 to 2.16 per book, across the nine books. The impact of 
the Story Friends treatment did seem to have a smaller im- 
pact, however, as children progressed through the treatment. 
Importantly, the pre-intervention treatment group scores 
did not differ from either the pre-intervention comparison 
group scores or the postintervention comparison group 
scores across any of the books. 

Finally, in order to evaluate the size of the effect of 
the Story Friends treatment, we calculated Cohen’s ds com- 
paring the postintervention scores from the children in Story 
Friends treatment to the postintervention scores of children 
in the control group, across each book in the treatment. 
Cohen’s ds ranged from 1.04 to 2.77 across the nine books, 
with a mean d = 1.83. Cohen (1988) suggested that ds of 
0.20, 0.05, and 0.80 represent small, medium, and large ef- 
fects, respectively. Consequently, the Story Friends treat- 
ment had a large effect on vocabulary learning across all of 
the books. 


Moderation in the Treatment Group 


Because there was clear evidence of an impact of the 
Story Friends treatment on word learning, we next explored 
potential moderating variables within the treatment group. 
Specifically, the impact of number of instructional listens 
during the intervention and CELF-P2 scores were evaluated 
as moderators of the association between book and word 
learning, again using an MLM approach. As reported in pre- 
vious research (Goldstein et al., 2016), there was no main ef- 
fect of instructional listens on word learning, F(1, 331) = 3.58, 


Table 5. Significance tests for fixed effects on word score. 


Factor df F p 

Pre PPVT-4 1,78 14.26 <.01 
Group 1,19 28.65 <.01 
Book 8, 1282 5.45 <.01 
Group x Book 8, 1282 2.52 01 
Time 1, 1293 548.26 <.01 
Group x Time 1, 1293 244.63 <.01 
Book x Time 8, 1279 2.46 01 
Group x Book x Time 8, 1279 2.88 <.01 


Note. PPVT-4 = Peabody Picture Vocabulary Test—Fourth Edition 
(Dunn & Dunn, 2007). 
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Figure 3. Average word points earned by participants across books, by group at pre- and posttreatment. Error 


bars are 95% confidence intervals. 
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p = .06, and there was no interaction between instructional lis- 
tens and book, F(8, 303) = 1.41, p = .19. For the CELF-P2, 
there was a significant main effect on word points, F(1, 32) = 
9.81, p < .01. The unstandardized regression coefficient asso- 
ciated with that effect, b = 0.10, SE = 0.03, indicated that ev- 
ery l-point increase in CELF-P2 scores was associated with 
approximately 0.10 increase in word points across books. The 
interaction between CELF-P2 and book was not significant, 
F(8, 278) = 1.55, p = .14. 


Retention of Vocabulary in the Treatment Group 


To evaluate the retention of vocabulary knowledge 
following intervention, we tested children’s knowledge of 
six vocabulary words from Unit | at the end of Unit 3, ap- 
proximately 2 months after words were taught. The word 
scores for pre-intervention, postintervention (Unit 1 posttest), 
and retention (Unit 3 posttest) for participants in the treat- 
ment group were submitted to a one-way within-subjects 
analysis of variance. The results indicated there was a signifi- 
cant difference between the three time points, F(2, 114) = 
37.21, p < .01. A Tukey’s honestly significant difference 
post hoc analysis indicated that pre-intervention word score 
(M = 0.45, SD = 0.98) was significantly lower than post- 
intervention word score (M = 6.56, SD = 3.66) and the reten- 
tion word score (M = 5.13, SD = 4.16). Importantly, there 
was no significant difference between the postintervention 
word score and the retention word score, indicating that 
participants generally retained the word knowledge gained 
during the intervention after its completion. 


Dosage and Implementation Fidelity 


To describe dosage and implementation fidelity, 
teachers were asked to complete daily attendance and pro- 
cedural checklists. However, teachers did not always com- 
plete these forms. Across books and classrooms, 8% of 
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attendance data were missing, meaning that there was no 
indication whether a child had listened to that book. A 
similar amount of data were missing in treatment and com- 
parison classrooms (7% and 9%, respectively), and most 
data were missing for review books (28% missing). Data 
from the procedural checklists were missing for two reasons: 
Either teachers did not complete the form, or teachers re- 
corded attendance but did not complete the procedural 
fidelity checklists. Across books and classrooms, 18% of 
procedural checklists were missing, more from treatment 
classrooms (26% missing) than from comparison class- 
rooms (11%). 

We calculated dosage based on the available data. 
Because so much of the missing data were from review 
books, we report the means for instructional books and 
review books separately. Overall, dosage was high in both 
groups. In treatment classrooms, the average number of lis- 
tens for instructional books per unit was 7.28 of a possible 
9 (SD = 1.81, range: 2.67—9) and the average number of lis- 
tens for review books was 1.86 of a possible 3 (SD = 1.04, 
range: 0-3). In comparison classrooms, the average number 
of listens per unit was 8.13 for instructional books (SD = 
1.00, range: 4.67-9) and 2.67 for review books (SD = 0.68, 
range: 0-3). Across units, the number of listens for instruc- 
tional books remained essentially the same for both groups 
and did not differ significantly. The treatment group aver- 
aged 7.20, 7.40, and 7.14 listens across the units; the com- 
parison group averaged 7.97, 8.00, and 8.13 listens across 
the units. 

We calculated implementation fidelity using available 
procedural checklists. Overall, there was a high level of 
adherence to the program. In treatment classrooms, aver- 
age fidelity was 95% (range: 67%-100%). In comparison 
classrooms, average fidelity was 97% (range: 33%—100%). 
Implementation fidelity also was documented during obser- 
vations by research staff. These observations were planned 
to occur | time per unit in each classroom, but slightly 
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fewer observations occurred (average: 2.7 for treatment 
classrooms, 2.3 for comparison classrooms). Implementa- 
tion fidelity during observations was high; average fidelity 
across observations was 90% (range: 60%-100%) in com- 
parison classrooms and 96% (range: 67%—100%) in com- 
parison classrooms. Across all classrooms, only two of the 
six items on the observations checklist received scores of 
80% or lower (i.e., facilitator used headphones, and envi- 
ronment was quiet). 

During observations, research staff documented in- 
struction provided by the teachers, use of positive feedback, 
and interruptions to the listening center. In comparison 
classrooms, teachers were never observed to provide in- 
struction on any of the Story Friends vocabulary words. 
In treatment classrooms, teachers occasionally provided 
additional prompts or instruction of target words beyond 
what was provided in the prerecorded stories (e.g., “ You’re 
going to be disappointed if we don’t go outside. Right?”), this 
was noted in about half of observations. Positive reinforce- 
ment and/or redirection of off-task student behavior was 
noted to occur as needed during the majority of observa- 
tions. Interruptions (e.g., a child outside of the listening 
center interrupted teacher to ask a question) were noted in 
about a quarter of observations. 

Some variations in the delivery of listening center were 
observed. In three treatment classrooms, teachers did not 
wear headphones consistently. On rare occasions, teachers 
were not able to stay in the listening center with the small 
group because they did not have assistance in the classroom, 
making it difficult to manage children in the other centers. 
In addition, teachers in two classrooms were observed to 
play the book audio using a speaker so that all children in 
the classroom could listen to the story together. Variations 
such as these were expected as teachers integrated the Story 
Friends program into typical classroom routines. 


Social Validity Results 


Teacher Social Validity Survey Results 

At the end of the study, we collected social validity 
surveys from 18 of the 24 classroom teachers, 10 from treat- 
ment classrooms and eight from comparison classrooms. 
Teachers responded to items related to feasibility and accept- 
ability of implementation using a 1-6 scale (1 = strongly dis- 
agree to 6 = strongly agree) and to open-ended questions. 
Overall, results indicated high social validity. Teachers indi- 
cated that the Story Friends program was easy to implement 
(average rating of 5.7 out of 6) and fit well within a rotation 
of center activities (5.3 out of 6). Teachers were highly moti- 
vated to use Story Friends in the future (average rating of 
5.8 out of 6) and felt that the program was a good way to 
address language delays in young children (average rating: 
5.4). Teachers in the treatment classrooms were mostly pos- 
itive about classroom and home strategies as well. Teachers 
sometimes used the review charts when words were practiced 
in the classroom (3.9 out of 6) and sent word necklaces (av- 
erage rating: 5.3) and stickers home weekly (average rating 
of 5 out of 6). Teachers in both conditions did not rate the 


use of Bloomz highly. Although research staff assisted in 
the setup of Bloomz, teachers reported that their parents 
rarely interacted with the application (average rating of 

2.9 out of 6). 

Responses to open-ended questions were generally posi- 
tive. Teachers reported that their students loved the stories, 
but some teachers in the comparison classrooms reported 
that students were bored or distracted during the third listen 
of the book. Teachers noted several benefits of the program 
including learning new vocabulary words and learning con- 
cepts of print and book conventions. Children had more 
opportunities for conversations around new words and the 
storybook contexts and provided opportunities to practice 
answering questions about books, which helped to improve 
their listening and comprehension skills. Teachers liked that 
each book used relatable contexts for preschoolers (i.e., first 
day of school, going to the doctor), the format of the listen- 
ing center. One teacher reported the program helped her 
with literacy planning for her lesson plans. 

Teachers in the treatment condition extended vocabu- 
lary instruction to other parts of the school day and encour- 
aged their students to use the words as well. For example, 
several teachers tried to incorporate the words in whole 
group settings, while children were at learning centers, dur- 
ing lunch, and playground time (1.e., Jf’s time to clean up, 
this area is filthy; You’re preparing lunch for us like in the 
book, yow’re getting it ready). Interestingly, several teachers 
indicated they did not use or would forget to use the vocab- 
ulary review chart even though practice and review oc- 
curred throughout the day. They also commented on the 
use of the home extension materials, indicating that chil- 
dren were excited to wear the necklaces and stickers home 
and that parents liked them as well. Three teachers noted 
that parents did not use the home materials or, at least to 
their knowledge, were unaware of their parents’ response 
to the home extension program. Comparison teachers noted 
students loved the storybook characters and would talk 
about the characters and the stories outside of the listening 
center. For example, one teacher had students who would 
“play” Bobby Bear on the playground. 

Overall, teachers were happy with the program and 
only recommended minor changes such as using a device 
other than the mp3 player because it was difficult to navi- 
gate, implementing the small-group listening center with 
all of the children in their class so all could benefit from 
the program. Teachers noted that parents’ use of Bloomz 
was very limited. Although they thought it was a useful 
tool, most were unable to get any parent to participate in 
the application. One teacher would have liked to have the 
program in both English and Spanish so all her students 
could participate. 


Parent Social Validity Survey Results 

We received 11 social validity surveys back from par- 
ents (out of a possible 84), five from five different treatment 
classrooms and six from four different comparison class- 
rooms. Although the return rate was low (13%), those who 
returned the surveys provided valuable feedback. Parents 
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were motivated to use the home extension materials with 
their children (4.8 out of 6) and felt that the program was a 
good way to address children’s language needs (average rating 
of 4.5). They were able to implement the program at home in 
a variety of family activities (average rating of 4.7), and the 
time required to do so was adequate (average rating of 4.6). 
Parents with children in treatment classrooms reported 
that their children sometimes came home with vocabulary 
necklaces or wearing stickers (average rating of 2.4 out of 3). 
Despite the low rating, parents commented that they felt the 
necklaces were effective and used them to practice the words 
each week. Parents reported that they rarely use Bloomz 
(rating of 1.4 out of 3) nor did they access the short prac- 
tice videos (average rating of 1.4 out of 3). Furthermore, 
five parents reported that they did not have access or did not 
know about Bloomz and the parent videos. Overall, parents 
were pleased with the program but recommended small 
changes to include a bilingual component or small story- 
books to accompany the vocabulary necklaces so parents 
could practice reading the books with their children. 


Discussion 


The purpose of the current study was to examine the 
effects of the revised Story Friends program on the vocabu- 
lary knowledge of preschool children. The program was 
implemented by educational staff in preschool classrooms 
that served families with low-income, and participants were 
children with limited oral language skills. There were strong 
treatment effects on the measure of vocabulary knowledge. 
Prior to intervention, children rarely knew any of the target 
vocabulary words (pretest average of less than 4% of words). 
At posttest, children in the treatment classrooms demon- 
strated substantial gains, with average posttest scores of 
30.25 of a possible 72 word points across the 13 weeks of 
the program, representing knowledge of 42% of taught 
words or approximately 15 of 36 words. Findings from the 
retention test indicate that children generally maintained 
their knowledge of taught vocabulary words approximately 
2 months after instruction. In contrast, very little learning 
was observed in comparison classrooms with average post- 
test scores of just 7.82 word points (11% of taught words). 

Findings indicate that the revisions to Story Friends 
were successful in increasing the effects on children’s vocabu- 
lary knowledge. Although we did not conduct a direct com- 
parison of the revised program and the original program, 
we can compare the findings of this study to our examination 
of the previous version of Story Friends (Goldstein et al., 
2016). In that study, children in the treatment classrooms 
had posttest averages representing 28.3% of taught words 
or approximately five of 18 words per series. In the current 
study, children learned a higher percentage of taught words 
(42%), and we taught twice the number of words. Thus, 
children learned almost 3 times as many words (15 of a pos- 
sible 36) in the current study. Postintervention vocabulary 
knowledge was higher than similar studies by other research 
groups (27% in Penno et al., 2002). In Storkel et al. (2017), 
kindergarteners with developmental language disorders 
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who received 36 exposures learned five of 30 taught words 
(17%). The higher percentage of words learned in the current 
study might be explained by increased treatment intensity; 
children who received the intended dosage of the small- 
group listening centers (three listens to each instructional 
book and three listens to the review book) would have re- 
ceived at least 48 exposures to each target vocabulary word 
in embedded lessons, with potential additional exposures in 
the classroom and at home. As hypothesized, by teaching 
more words and by creating additional opportunities for 
review and practice in the classroom and at home, children 
in the treatment classrooms made substantial gains in vocab- 
ulary knowledge. 

Children learned the most words in Unit 1 (M = 11.73 
of a possible 24 word points, SD = 6.96); posttest scores 
were lower in Unit 2 (M = 10.75, SD = 7.33) and Unit 3 
(M = 7.77, SD = 6.29). We anticipated that dosage (i.e., 
number of listens) might contribute to the decline in learn- 
ing, but this was not the case. It may be that motivation 
and engagement of teachers and children decreased across 
the study. Near the end of the school year, the demands on 
teachers’ time may increase and changes to daily schedules 
are more frequent. We observed that children were less 
compliant during testing as the study continued; the novelty 
of a visit from a new adult seemed to wear off. Another 
possibility is that there were differences in word difficulty 
across the Forest Friends series. We used consistent criteria 
to select vocabulary words, but it may be that words pre- 
sented later in the series were harder to learn. In future 
studies, it will be important to carefully monitor variables 
that relate to engagement and implementation. 

As in previous studies of Story Friends, there was 
considerable variability in learning among participants; 
25% of the variance in word points was explained by the 
child. At each unit, some children in treatment classrooms 
learned no new words, whereas others learned nearly every 
word taught. The moderation analysis indicated that chil- 
dren with higher pre-intervention language scores learned 
more words in Story Friends than children with lower 
scores. However, even children with low pre-intervention 
language skills made substantial gains: The 13 partici- 
pants in the treatment group with PPVT-4 standard scores 
below 85 gained an average of 15 word points, represent- 
ing approximately one new word per book. Thus, pre- 
intervention language ability did not fully predict response 
to intervention. However, the wide variability in children’s 
learning highlights the importance of quickly and effi- 
ciently identifying those children who are not learning. 
In our previous RCT, learning during the first few weeks 
of intervention was a sensitive benchmark to identify poor 
responders (Kelley et al., 2018). To effectively implement 
Story Friends as part of a multitiered system of support, it 
will be important to efficiently identify children who need 
additional support. Children who do not respond to the 
small-group listening center might benefit from increased 
intensity. Next steps in this line of research would include 
testing the effects of an individualized variation of Story 
Friends for struggling learners. 
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In comparison classrooms, children were exposed to 
the target vocabulary words in the context of the prere- 
corded storybooks but learned few vocabulary words. This 
finding is consistent with our previous studies and with a 
larger body of evidence on vocabulary learning; simply 
exposing children to challenging vocabulary words is not 
sufficient to produce meaningful change in vocabulary knowI- 
edge (Beck & McKeown, 2007; Goldstein et al., 2016). 


Implications for Vocabulary Instruction 
in Preschool Classrooms 


Results of the current study provide guidance on the 
appropriate number of academic vocabulary words that 
can be targeted in explicit instruction during shared story- 
book reading and reinforced by teachers and families. The 
revised Story Friends program taught a total of 36 chal- 
lenging vocabulary words, four words per book and twice 
as many words as in previous versions. We determined 
that four words could be successfully included in automated 
listening centers in a series of single-case experimental de- 
sign studies (Peters-Sanders et al., 2020); findings of the cur- 
rent study reinforce the decision to include the additional 
vocabulary targets. Many of the teachers we worked with 
had the habit of choosing a single challenging vocabulary 
word per book; we suggest that vocabulary learning in pre- 
school classrooms can be maximized by teaching at least 
four academic vocabulary words per week. 

We hypothesize that the additional practice opportu- 
nities provided by teachers and families contributed to in- 
creased learning in the revised Story Friends program. In a 
previous study (Seven et al., 2020), children demonstrated 
substantial learning of vocabulary words when the only in- 
struction came from classroom strategies (among typically 
developing children who did not participate in listening 
centers). Among the children receiving small group instruc- 
tion, half the words were subject to classroom practice 
strategies, which clearly boosted learning in the majority 
of participants. Similar to the Seven et al. study, informal 
communication with teachers and observations of our re- 
search staff in the current study indicated that, when teachers 
frequently used classroom strategies, all children in those 
classrooms benefited from the practice strategies and learned 
many of the targeted vocabulary words. Classroom strategies 
were designed to promote review of previously taught words 
and may have contributed to the maintenance of vocabulary 
knowledge on the retention test. We suggest that children’s 
learning of academic vocabulary can be increased by the use 
of feasible strategies for practice and review across the school 
day and at home. 


Limttations and Future Directions 


As with any study conducted in classroom settings, 
there was some variability in the ways in which the program 
was implemented. Our criteria for fidelity of implementation 
focused on the delivery of the automated listening center. We 
planned for and expected variation in the use of classroom 


and home strategies. We encouraged teachers to use class- 
room strategies and materials to embed practice opportunities 
in a variety of settings across the school day, which made 
precise measures of implementation difficult. Instead, we 
relied on observations and social validity surveys to answer 
our research question about feasibility. In some classrooms, 
teachers made use of the materials and strategies as intended 
by practicing target vocabulary words in meaningful class- 
room contexts, using posters and cards as prompts and re- 
view opportunities. Some teachers used target vocabulary 
words in creative ways, for example, as journal prompts. In 
other classrooms, little use of classroom strategies was ob- 
served: Posters and related materials were not readily visi- 
ble in the classroom, and teachers rarely reported using the 
target vocabulary words. Although we anticipate that these 
differences in implementation would relate to differences 
in learning, we saw very few differences in learning across 
classrooms. The classroom ICC was low (6.3%), suggest- 
ing that differences in implementation of the classroom 
strategies did not have a large impact on learning. In future 
studies, we can add detail to our understanding of class- 
room implementation by increasing the frequency or dura- 
tion of observations and by adding feasible self-reporting 
strategies. 

We did not anticipate such little interest in Bloomz. 
In our previous development studies, parental use of mes- 
saging applications was higher, even when teachers were re- 
sponsible for coordinating the communications (Soto et al., 
2020). When Bloomz was unsuccessful, we adapted our 
approach by sharing the training videos on i1Pads and at par- 
ent conferences and by sending e-mail messages. However, 
we were unable to track the extent to which teachers sent 
e-mail messages or the use of home strategies with indi- 
vidual families, and the return rate of the family social 
validity survey was low. The challenge of effective family 
engagement is not unique to our project; other studies 
have reported similarly low rates (Sheridan et al., 2011). 
Continued, innovative effort in this area is necessary to 
produce interventions that are acceptable and feasible for 
use with at-risk families. 

The current study provides an example of a hybrid 
study (Curran et al., 2012) in which we examined both treat- 
ment efficacy and implementation of the Story Friends in- 
tervention package. Findings provide strong evidence of 
treatment efficacy. Although observations and social valid- 
ity data were a rich source of information on the implemen- 
tation of the small-group listening centers, the information 
on implementation on classroom and home practice com- 
ponents is far less complete. We planned for ways to gather 
information about teachers’ and families’ use of materials 
(e.g., pictures of Story Friends chart, communication via 
Bloomz). Although we would have gathered more informa- 
tion from teachers and parents, this does not mean that 
their implementation was poor. Indeed, it varied from poor 
to excellent. In future studies, we will examine feasible, 
efficient ways for teachers and families to report use of 
strategies. Nevertheless, these findings contribute to our 
understanding of the characteristics of interventions that 
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are feasible for implementation in authentic education 
settings. These hybrid designs can help to accelerate the 
application of research findings into practical settings 
(Schliep et al., 2017). 


Conclusions 


Findings indicate that a carefully designed interven- 
tion can produce robust vocabulary learning by high-risk 
children. Children in the Story Friends treatment classrooms 
were able to define an average of 42% of taught words, all 
selected to be challenging and academically relevant, after 
participating in 13 weeks of a low-dose intervention. The 
program was implemented in early childhood classrooms 
by educational staff who received minimal training (1 hr), 
indicating that the program is feasible for implementation 
in authentic educational settings. Although large treatment 
effects were observed, averaging d = 1.83, the iterative de- 
velopment process should continue to refine classroom and 
home strategies that can increase opportunities for review 
and practice. 
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Group comparisons for children with complete data and children with missing data 


Complete data Missing data 
Variable M SD M SD t p d 
IGDI PN 47.4 2.09 47.14 2.14 -0.33 74 0.07 
CELF-P2 80.71 9.03 74.36 9.25 -2.16 <.05 0.69 
Pre PPVT-4 88.4 7.96 90.43 7.39 -1.10 27 0.24 
Pre word score 0.40 0.42 0.31 0.38 -1.02 31 0.23 
Post word score 2.08 1.99 2.06 1.71 -0.05 .96 0.01 
Post PPVT-4 92.71 9.41 94.25 11.84 -0.03 5 0.16 
Note. IGDI PN = Individual Growth & Development Indicators Picture Naming (Bradfield et al., 2014); CELF-P2 = Clinical 


Evaluation of Language Fundamentals Preschool—Second Edition (Wiig et al., 2004); PPVT-4 = Peabody Picture 
Vocabulary Test—Fourth Edition (Dunn & Dunn, 2007); Pre word score = pretest score per book out of 8 possible points; 
Post word score = posttest score per book out of 8 possible points. 
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